Road curvature generation in real-world images as a method of data augmentation

ABSTRACT

Disclosed embodiments provide systems and methods for transforming real-world images for augmentation of image data for improvement of training data sets utilized by machine learning algorithms to provide autonomous and/or assistive driving functionality.

CROSS-REFERENCE TO RELATED APPLICATIONS

None

FIELD

Disclosed embodiments relate to method operations and equipment for usein transforming real-world images for augmentation of image data.

SUMMARY

In accordance with disclosed embodiments, systems, components, andmethodologies are provided to improve training data sets utilized bymachine learning algorithms to provide autonomous and/or assistivedriving functionality.

In accordance with at least some disclosed embodiments, systems,components and methodologies, real-world data is augmented withsynthetic data generated from the real-world data. Disclosed embodimentsprovide systems and methods for transforming real-world images foraugmentation of image data for improvement of training data setsutilized by machine learning algorithms to provide autonomous and/orassistive driving functionality.

Additional features of the disclosed embodiments will become apparent tothose skilled in the art upon consideration of disclosure providedherein.

BRIEF DESCRIPTION OF THE FIGURES

The detailed description particularly refers to the accompanying figuresin which:

FIGS. 1A and 1B illustrate examples of highway images for training ofmachine learning algorithms that facilitate autonomous and/or assistivedriving functionality in accordance with disclosed embodiments.

FIGS. 2A and 2B illustrate an example of highway images for training ofmachine learning algorithms with vehicle paths superimposed thereon inaccordance with disclosed embodiments.

FIG. 3 illustrates an example of a system and constituent components forperforming original image data acquisition and transformation of theoriginal data in accordance with the present disclosure.

FIG. 4 illustrates the operation of performing image data acquisitionand transforming for development of augmented training data sets fortraining of machine learning algorithms in accordance with the presentdisclosure.

DETAILED DESCRIPTION

The figures and descriptions provided herein may have been simplified toillustrate aspects that are relevant for a clear understanding of theherein described devices, systems, and methods, while eliminating, forthe purpose of clarity, other aspects that may be found in typicaldevices, systems, and methods. Those of ordinary skill may recognizethat other elements and/or operations may be desirable and/or necessaryto implement the devices, systems, and methods described herein. Becausesuch elements and operations are well known in the art, and because theydo not facilitate a better understanding of the present disclosure, adiscussion of such elements and operations may not be provided herein.However, the present disclosure is deemed to inherently include all suchelements, variations, and modifications to the described aspects thatwould be known to those of ordinary skill in the art.

Machine learning algorithms, often used in connection with autonomoustransportation vehicles, call for large quantities of data whendeveloping models by which autonomous and/or assistive drivingfunctionality is performed. The large quantities of data used duringdevelopment of these models should be diverse and balanced, but it isfrequently expensive to acquire this type of data from real-world datagathering.

Disclosed embodiments provide a technical solution for improving a dataimbalance problem for training machine learning models using datacollected on roadways, particularly controlled access highways and thelike. For example, often, highways and like roadways, include aproportionally greater quantity of straight roadway.

For the purpose of this disclosure, the term “roadway” includes anyroad, thoroughfare, route, or way on land between two places that hasbeen paved or otherwise improved to allow travel by a transportationvehicle including, but not limited to, a motor vehicle or othertransportation vehicle including one or more wheels. Thus, it should beunderstood that such roadways may include one or more lanes andjunctions with other roadways including on/off ramps, merge areas, etc.,that may be included in parkways, avenues, boulevards, freeways,tollways, interstates, highways, or primary, secondary, or tertiarylocal roads.

Thus, the term “highway,” as it is used herein refers to a specificsubset of roadways that typically include long stretches of straightroadway in greater proportion to curved or arcuate roadway. An exemplaryhighway, as contemplated by this disclosure, is a controlled accesshighway or autobahn. Therefore, the term “roadway” inclusively refers to“highways” in this disclosure.

For the purposes of this disclosure, the term “on-road localization” isused to refer to the ability to determine a position of thetransportation vehicle relative to the roadway, or a portion of aroadway such as a lane, on which the transportation vehicle istravelling.

With the further incorporation of autonomous and driverassistance-related transportation vehicle technologies, it is envisionedthat, in implementation, autonomous and/or assistive functionality willrely at least partially, and potentially fully, on on-road localizationperformed in an automated or semi-automated manner based on GlobalPositioning Service (GPS) data, data generated by a plurality of sensorslocated on-vehicle, and machine learning algorithms and/or neuralnetworks operatively coupled to the plurality of sensors and/or GPS forprocessing and interpreting such data to facilitate on-roadlocalization. As explained below, various conventional approaches areknown for developing and training machine learning models to executeon-road localization using various different types of sensors andvarious different types of training data sets. However, each of theseapproaches has deficiencies with regard to the quality/amount of data tobe acquired and/or processed during build-up of the machine learningmodel(s) for real-world use in real-time to safely controltransportation vehicle travel.

For the purposes of this disclosure, the phrase “autonomous and/orassistive functionality” refers to functionality that enables thepartial, full, or complete automation of vehicular control ranging andencompassing what has presently come to be known as the five levels ofdriving automation. Thus, it should be understood that autonomous and/orassistive functionality refers to operations performed by a vehicle inan automated manner by on-vehicle equipment or the output of alerts,prompts, recommendations, and/or directions to a user, wherein theseoutputs are generated in an automated manner by on-vehicle equipment.Moreover, autonomous and/or assistive functionality may include driverassistance functionality (level one) wherein on-vehicle equipmentassists with, but does not control, steering, braking and/oracceleration, but a driver ultimately controls accelerating, braking,and monitoring of a vehicle surroundings.

It should be understood, therefore, that such autonomous and/orassistive functionality may also include lane departure warning systemswhich provide a mechanism to warn a driver when a transportation vehiclebegins to move out of its lane (unless a turn signal is on in thatdirection) on freeways and arterial roads. Such systems may includethose that warn the driver (Lane Departure Warning) in response to thevehicle is leaving its lane (visual, audible, and/or vibration warnings)and, if no action is taken, automatically take operations to ensure thevehicle stays in its lane (Lane Keeping System).

Likewise, autonomous and/or assistive functionality may include partialautomation (level two), wherein the transportation vehicle assists onsteering or acceleration functions and correspondingly monitoringvehicle surroundings to enable a driver to disengage from some tasks fordriving the transportation vehicle. As understood in the automotiveindustry, partial automation still requires a driver to be ready toassume all tasks for transportation vehicle operation and also tocontinuously monitor the vehicle surroundings at all times.

Autonomous and/or assistive functionality may include conditionalautomation (level three), wherein the transportation vehicle equipmentis responsible for monitoring the vehicle surroundings and controlssteering, braking, and acceleration of the vehicle without driverintervention. It should be understood that, at this level and above, theon-vehicle equipment for performing autonomous and/or assistivefunctionality will be interfacing with or include navigationalfunctionality so that the components have data to determine where thevehicle is to travel. At level three and above, a driver istheoretically permitted to disengage from monitoring vehiclesurroundings but may be prompted to take control of the transportationvehicle operation under certain circumstances that may preclude safeoperation in a conditional automation mode.

Thus, it should be understood that autonomous and/or assistivefunctionality may include systems that take over steering and/ormaintain the transportation vehicle relatively centered in the lane oftraffic.

Likewise, autonomous and/or assistive functionality may include highautomation (level four) and complete automation (level five), whereinon-vehicle equipment enable automated steering, braking, andaccelerating, in response to monitoring of the surroundings of thevehicle in an automated manner without driver intervention.

Therefore, it should be understood that autonomous and/or assistivefunctionality may require monitoring of surroundings of a vehicleincluding the vehicle roadway as well as identification of objects inthe surroundings so as to enable safe operation of the vehicle inresponse to traffic events and navigational directions, wherein thatsafe operation requires determining when to change lanes, when to changedirections, when to change roadways (exit/enter roadways), when and inwhat order to merge or traverse a roadway junction, and when to use turnsignals and other navigational indicators to ensure othervehicles/vehicle drivers are aware of upcoming vehicle maneuvers.

Further, it should be understood that high and full automation mayinclude analysis and consideration of data provided from off-vehiclesources in order to make determinations of whether such levels ofautomation are safe. For example, autonomous and/or assistivefunctionality at such levels may involve determining the likelihood ofpedestrians in the surroundings of a transportation vehicle, which mayinvolve referencing data indicating whether a present roadway is ahighway or parkway. Additionally, autonomous and/or assistivefunctionality at such levels may involve accessing data indicatingwhether there is a traffic jam on the present roadway.

Conventional transportation vehicle navigation systems, as well as,conventional autonomous vehicles use GPS technology for their on-roadlocalization. However, a deficiency of conventional use of globallocalization using GPS is that localization is limited to a certainlevel of accuracy, more specifically 5-10 meters in the best case (whichtypically requires a geographical area that provides an unobstructed,open view to the sky). Moreover, lower accuracy is much more likely ingeographical areas that include relatively large buildings, trees, orgeographic contours such as canyons. This is because GPS based locationservices require signals from GPS satellites. Dense materials, forexample, rock, steel, etc.), tall buildings, and large geographicalterrain features may block or degrade GPS signals.

Therefore, GPS has been conventionally combined with local landmarks foron-road localization, for example, lane markings to improve the abilityfor a vehicle with autonomous and/or assistive vehicle functionality toaccurately perform on-road localization. Conventionally, these locallandmarks have been detected and identified from camera images or sensordata from other sensors obtained by one or more cameras/sensors locatedon-vehicle. For example, it has been conventionally discussed to combineGPS data with data collected from front view cameras and LiDAR, and,even data generated by ground penetration radar. In addition, it hasbeen discussed that there is utility for such cameras to extract areduced feature presentation of roadway characteristics from on-vehiclecameras to generate data indicating roadside patterns that may beanalyzed to perform on-road localization. Machine learning algorithms,and models developed thereby, facilitate the combination and operationalutilization of the various data inputs, including camera images, toprovide autonomous and/or assistive functionality.

Disclosed embodiments are based on a recognition that recent autonomousvehicle traffic accidents provide evidence that there is a technical andreal-world need to increase the robustness and/or reliability of machinelearning models that govern autonomous and/or assistive functionality;particularly, in scenarios that occur less frequently and/or requiremore significant reactionary measures from the systems enabling theautonomous and/or assistive functionality and controlling travel oftransportation vehicles.

Furthermore, autonomous and/or assistive functionality often leveragesmachine learning models. Generally, development of autonomous and/orassistive functionality may take one of two different approaches. Afirst approach emphasizes the input of modular systems (e.g., a sensorfusion module, a scene understanding module, a dynamics predictionmodule, a path planning module, and a control module). A second approachis an end-to-end methodology whereby sensory data may be mapped directlyto control signals at the instruction of a computer algorithm.

Machine learning may be heavily utilized in both the first and secondapproaches. Machine learning models are trained with a dataset that maybe referred to as “training data”. The ability and operational value ofmachine learning algorithms is indirectly affected by the quality andquantity of the training data. Still further, the training data must begathered or generated, dependent upon numerous considerations andcomparisons associated with the availability and expense of both thegeneration and collection of particular data subsets of the trainingdata. In exemplary machine learning models, real-world data is gatheredfor training. Further, in accordance with this example of a machinelearning system underpinning autonomous and/or assistive functionalityof transportation vehicles, training data may be collected by sensorsmounted in test vehicles driven by test drivers for extended periods oftime.

Machine learning models may be built and trained with the goal offacilitating operation thereof, in this case autonomous and/or assistivefunctionality, with all types of inputs spanning an input domain of themodel (e.g., all roadway circumstances and conditions). For asimplified, but analogous, example, if a machine learning algorithm isaimed at detecting dogs in pictures, then the algorithm aims to be ableto detect all breeds of dogs. However, dog breeds may be quite visuallydifferent from one another. Thus, to develop the machine learningalgorithm adequately, such that all dogs are recognized, the trainingdata should be well balanced and diverse. Referring still to ahypothetical dog detecting algorithm, the implication for the trainingdata is that roughly equal proportions of images of all dog breedsshould be present therein in order to build a robust machine learningmodel.

Thus, regarding autonomous and/or assistive functionality during highwaydriving, a data imbalance problem within training data sets presentsmultiple challenges. First, most segments of highway are either straightor very slightly curvy, and only a small fraction thereof includeconsiderable curvature. Therefore, a machine learning algorithm trainedwith unfiltered and/or unaltered data collected from extended periods ofhighway driving will have been over-trained on straightaways andunder-trained on curved road segments. As a result, a transportationvehicle utilizing autonomous and/or assistive functionality relying onthis example machine learning model would likely perform poorly onarcuate highway segments, while still not realizing significantimprovement over a baseline in performance on straightaway highwaysegments.

Described with reference to FIGS. 1-4, systems and methods of trainingdata transformation address natural imbalances in training data gatheredfor training a machine learning algorithm utilized in the performance ofautonomous and/or assistive functionality.

Conventionally, two strategies are available for addressing the presenceof a data imbalance between straight highway/roadway data 104 and curvedhighway/roadway data within a training data set. First, a machinelearning algorithm may be trained by a simulation. During simulation,the proportion of straightaway and curved road segments experiencedalong the simulated highway may be balanced as desirable. In otherwords, the entire body of training data may be customized. This approachmay represent significant expense. For example, teams of engineers andartists are needed to build a simulator that mimics real-world highwaydriving, and the mileage and scenery diversity specification for thetraining data would add to the cost of any developed simulator.

It may be useful, instead, to generate diverse roadway data withoutdeveloping a simulator. Still further, machine learning models trainedwith simulated data may not necessarily perform well in the real world.Simulated data, however realistic looking, may be simply not the same asreal-world data. There is no a priori guarantee that this simulatedapproach would produce a robust machine learning model.

In contrast, in accordance with disclosed embodiments, systems andmethodologies may generate synthetic data that is more photorealisticand more adaptable to real-world problems because the disclosedembodiments draw from real-world data.

However, when training with real-world data gathered by driving alonghighways is desired, then a portion of the straight highway data may beremoved from the training data set to develop a more diverse trainingdata set according to conventional approaches. Similarly, the curvedhighway data may be artificially replicated, perhaps numerous times,until the training data set comprises a desired ratio of the straighthighway data to the curved highway data. This approach also hasdisadvantages because removing data is wasteful. Data collection isexpensive. Removing a portion of straight road driving data is removes arelatively large portion of all data collected. Alternatively, repeatingthe curved road data numerous times such the amount thereof matches thestraight road data results in decreased within-category diversity in thecurved road data. Machine learning algorithms thrive models are developwith diversity and balance of training data. Artificially replicatingone type of data to achieve data type balance may solve the dataimbalance problem, but such actions create another data imbalanceproblem (e.g., that one category of training data is much more diversethan another category). This may result in the algorithm performingbetter with one type of input than another.

In accordance with disclosed embodiments, and as illustrated in FIGS.1A-1B, straight highway data 104 may comprise data describing one ormore straight highway segments 112, and the curved highway datacomprises data describing one or more curved highway segments 114 inaddition to other transformed image data. Accordingly, systems andmethods configured in accordance with disclosed embodiments may generatenew data in the curved highway segment category and may bring thatcategory to the same level of diversity as the straight highway segmentcategory.

Systems and/or methodologies of the presently disclosed embodiments maytransform images of the one or more straight highway segments 112 intoimages of one or more synthetic curved highway segments 114 withcomputer vision algorithms and/or processes.

FIG. 1A depicts an original image 116 of an example one of the straighthighway segment(s) 112 captured by a camera 118 (see FIG. 3) mounted toa transportation vehicle 120 (e.g., a car, once again, see FIG. 3). FIG.1B depicts a transformed arcuate image 122 of an example of the one ormore synthetic curved highway segments 114. In the transformed image 122of FIG. 1B, the original image 116 (FIG. 1A) is modified such that theroadway shown therein is imparted a curvature of 1/300 meters.Hereinafter, the term “curvature” is meant to refer to the reciprocal ofradius such that a smaller curvature implies a larger radius,relatively.

In FIG. 1B, a curvature of 1/300 m indicates that the illustratedexample of one of the synthetic curved highway segment(s) 114 comprisesa radius of 300 meter, should the segment of roadway be extended tocomplete a circle.

In accordance with disclosed embodiments and shown in FIG. 4, a method400 for transforming the straight highway image 116 into the transformedarcuate image 122 may include: (i) identifying a horizon line 402; (ii)identifying direction information 404; (iii) processing pixels based onthe assumption that highways are relatively flat 406; and (iv)generating synthetic curved highway segments 408 from informationderived during the previous operations and extrinsic knowledge of thecamera used during a data gathering operation 410.

As illustrated in FIG. 3, and as part of identifying a horizon line,first, the original, straight highway images 116 may be acquired by acamera 118 disposed on or within the transportation vehicle 120. A lineof horizon 124 may be identified within the image(s). Numerous horizondetection algorithms may be utilized to assist in identifying thehorizon line 124 shown in FIGS. 1A-2B. Thereafter, a plurality of pixels126 disposed below the horizon line 124 may be identified ascorresponding to the straight highway segment 112. Thus, disclosedembodiments contemplate approximating that all pixels below the horizonline 124 represent the straight highway segment 112. Also, in accordancewith disclosed embodiments, additional techniques, such as identifyingroadway lines, may be utilized to increase the accuracy of a region ofpixels belonging to the roadway.

Subsequently, camera model and camera parameters adhered to during imagecollection may be used to inform an operation of transforming each ofthe highway pixels 126 to a light ray vector described by cameracoordinates. This is information described in image space, wherein eachof the light ray vectors indicates a direction associated with a portionof the straight highway segment 112 imaged by the individual pixel. Thedirection of each of the light ray vectors may be indicated relative tothe camera 118 but need not supply distance information. If an examplecamera has a 60-degree field of view, then it is possible to infer thatthe direction of an object is, for example, about 30 degrees to the leftof the forward direction. However, a 2D image does not impart to aviewer information on how far away the hypothetical object is disposed.To extend this example, the object may be small and nearby, or theobject may be relatively large and very far away. Thus, which one ofthese possibilities represents reality is not discernable from a 2Dimage alone.

As noted above, the original straight highway images 116 may be analyzedto reveal the direction of each of the highway pixels 126, and highwaystend to be relatively flat. Furthermore, a vertical distance 130 of thecamera 118 from a surface 132 of the highway is a known parameter of theextrinsic parameters of the camera model that facilitated collection ofthe original images 116. The extrinsic parameters define the locationand orientation of the camera with respect to the real-world frame ofreference. Extrinsic camera parameters may further include relativeposition and orientation of the camera with respect to the real-worldand/or the subject (e.g., the roadway). Intrinsic parameters of thecamera may also be important for data gathering and analysis asdescribed herein and may include parameters internal and fixed to aparticular camera/digitization setup. Intrinsic parameters allow amapping between camera coordinates and pixel coordinates in the imageframe and may include optical, geometric, and digital characteristics ofthe camera. Thus, this information informs an assumption that all thecolors in the highway pixels 126 come from a shared flat planeorthogonal to a vertical line connecting the camera 118 and the roadwaysurface 132. As a result, direction information for each of the highwaypixels 126 may be combined with the presence of each of the pixels 126on the flat plane at a known vertical distance 130 from the camera 118.This combination of information may be processed to determine a locationon the flat plane for each of the highway pixels 126. Accordingly, withthe addition of distance, the highway pixels 126 may be describedmathematically with spatial coordinates in physical space.

Once location information is determined for each of the highway pixels126, a curvature transformation may be applied to each point on the flatplane to develop a curved plane. Thereafter, the same camera model andextrinsic parameters previously leveraged to determine the light rayvector(s) for each of the highway pixels 126 may informre-transformation of the curved plane back into image space (e.g.,camera coordinates). Accordingly, the transformed arcuate image 122 ofthe one or more synthetic curved highway segments 114 may be generated.

Systems and methodologies provided in accordance with the disclosedembodiments may be utilized to perform training data transformation thatresults in blocks of black (or empty) pixels. Color and image dataoriginally present at image space corners of the highway pixels 126 maybe shifted left or right dependent upon on the curvature applied duringtransformation. As a result, corners of images of the one or moresynthetic curved highway segments 114 may appear without colorinformation. The presence of black or colorless corners is not desirablebecause the machine learning algorithm may be trained to identify theempty pixels as indicative of curvature, thereby decreasing theeffectiveness of the synthetic curved highway segments 114 within atraining data set. The presence of colorless pixels may be addressed byretaining the original colors at the image space location of theotherwise black pixels.

Systems and methodologies provided in accordance with the disclosedembodiments may utilize techniques from both the machine learning andcomputer vision fields. Further, systems and methodologies provided inaccordance with the disclosed embodiments may utilize an assumptionunique to the data collection method; specifically, disclosedembodiments may operate based on an assumption that the colors presentat image pixels belonging to a roadway may be treated as reflected offof the same flat plane. Thus, the curvature transformation process maythen be performed in a spatial coordinates system and a mapping betweenpixel coordinates and spatial coordinates may produce usable images forsupplementing a training data set. Further, the operation of coveringblack/colorless pixels caused by the curvature transformation processwith the original colors (e.g., retaining the original color informationeven after same information is spatially transformed away from theoriginal locations thereof) may, optionally, further improve the qualityof the training data set.

Systems and methodologies provided in accordance with the disclosedembodiments may be useful for developing autonomous and/or assistivefunctionality, such as when a neural network predicts a path of avehicle across approximately the next 30 meters for every image. Inparticular, the systems and methodologies provided in accordance withthe disclosed embodiments may generate images of one or more syntheticcurved highway segments 114 for augmenting the quantity of curvedhighway segments present in a gathered training data set. It is notablethat transforming a training image to add a certain curvature may bedesirably coupled with transforming data associated with the trainingimage. For example, when a vehicle path is associated with the trainingimage undergoing transformation, the corresponding vehicle path may betransformed to reflect the same curvature. Thus, the vehicle pathinformation may still follow the highway and/or a lane therein for thenext 30 meters for autonomous and/or assistive functionality. Whencurvature is added to an image without transforming the correspondingvehicle path, the original path will likely cross lanes or otherwiseincongruously indicate the vehicle path for the synthetic curved highwaysegments 114. This data augmentation technique may improve the accuracyand stability of a neural network model used to achieve autonomousand/or assistive functionality.

Referring now to FIGS. 2A and 2B, an original image 116 b and atransformed arcuate image 122 b are depicted. FIG. 2A is the originalimage 116 b of an exemplary one of the relatively straight highwaysegments 112 taken by the camera 118 mounted on the transportationvehicle 120. A first set of dots 138 indicates an actual path taken bythe driver over an upcoming 30 meters from the time the original image116 b was captured. Referring now to FIG. 2B, the transformed arcuateimage 122 b is shown after transformation imparts a high negativecurvature (e.g., curving to the right) to the highway segment. In FIG.2B, a second set of dots 140 indicates a new path transformed tocorrespond with the curvature imparted to the highway segment that adriver should take to maintain lane position on the synthetic curvedhighway segment 114 of the transformed arcuate image 122 b. The secondset of dots 140 of the curved path is obtained by applying the samecurvature transformation as is applied to the highway segment. FIG. 2Bfurther compares the first and second sets of dots 138, 140 respectivelyrepresenting the actual driven path and a path reflecting appropriateadjustment to fit the highway segment of the transformed arcuate image122 b.

Optionally, in addition, supervised learning, which is the most commonform of machine learning, involves enabling learning during a trainingphase based on a set of training data so as to enable the ability tolearn to recognize how to label input data for categorization. Deeplearning improves upon the supervised learning approach by consideringseveral levels of representation, in which every level uses theinformation from a previous level to learn more deeply. Deeperarchitectures of many stacked layers is one aspect, also ConvolutionalNeural Networks (CNNs) take into account 2D/3D local neighborhoodrelations in pixel/voxel space by convolution over spatial filters.

Supervised deep learning involves the application of multiple levels, orphases, of functional operations to improve understanding of resultingdata then fed into further functional operations. For example,supervised deep learning for classification of data into one or morecategories may be performed, for example, by performing feature learning(involving one or more phases of convolutions, Rectifier Linear Units(ReLUs) and pooling) to enable subsequent classification of sample datato identify learned features by application of a softmax function toenable differentiation between objects and background in the input imagedata. These operations may be performed to generate image class labelsfor classification purposes.

Likewise, supervised deep learning operations may be performed forregression by operating in parallel on Red Green Blue (RGB) image anddistance to ground/disparity map data by performing multipleconvolutions and joining the result through concatenation for subsequentprocessing. These operations may be performed to generate imageregression labels for subsequent use in analysis.

Moreover, supervised deep learning operations may be performed forsemantic segmentation by inputting RGB image data into a convolutionalencoder/decoder that may include multiple stages of convolution, batchnormalization (which does not only apply to segmentation but applies toother networks as well), ReLUs and pooling, followed multiple phases ofconvolution, batch normalization and ReLUs with upsampling. Theresulting data may then be processed by application of a softmaxfunction to provide output data with segmentation labelling for eachpixel. Thus, disclosed systems and methodologies for transforming imagedata may preserve real-world information for successful application ofthe above-outlined techniques.

Further, it should be understood that, although the disclosedembodiments may be utilized for the purposes of facilitating robustautonomous and/or assistive transportation vehicle functionalitygenerally, the disclosed embodiments may have particular utility inproviding that functionality when lane markings are obscured as a resultof weather conditions such as snow. In such conditions, lane markingsand roadside patterns conventionally used to assist in on-roadlocalization are obscured. For example, snow of any amount can obscurelane markings and road signs; additionally, snow of a large amount canalter the appearance of surroundings along a roadway to the point thatroadside patterns cannot be analyzed to provide additional data to becombined with GPS analysis for performing on-road localization.

In this regard, it should be understood that at least one embodiment mayinclude a feedback mechanism that determines the quantity and/or qualityof data produced and/or analyzed in the disclosed operations. Such afeedback mechanism may be used to selectively increase or decrease areliance on the transformed image(s) 122 in provisioning autonomousand/or assistive functionality. This may be implemented, for example, bydynamically weighting data that has or has not undergone transformation.It should also be understood that such a feedback mechanism may includecomparison with threshold values for maintaining at least minimumparameters to ensure safety for autonomous and/or assistivefunctionality operation.

Further it should be understood that a mechanism for dynamicallyweighting such data may be performed in one or more of variousconventionally known techniques that enable Sensor Data Fusion, forexample, using a Kalman Filter, processing performed based on thecentral limit theorem, Bayesian networks, the Dempster-Shafer theorem,CNNs or any of the other mathematical operations disclosed herein.

As explained above, disclosed embodiments may be implemented inconjunction with components of autonomous and/or assistive drivingsystems included in transportation vehicles. Thus, the utility of thedisclosed embodiments within those technical contexts has been describedin detail. However, the scope of the innovative concepts disclosedherein is not limited to those technical contexts.

Additionally, it should be understood that the presently disclosed meansfor analyzing image data depicting a roadway on which the transportationvehicle may comprise any combination of the sensors and functionalitydisclosed herein implemented in hardware and/or software to provide thedisclosed functionality.

Moreover, it should be understood that such assistive technology mayinclude but is not limited to what may have been conventionally termed aDriver Assistance System (DAS) or an Advanced Driver Assistance System(ADAS) implemented using hardware and software included in atransportation vehicle. These conventionally known systems assist thedriver in decision and control, but inevitably the decisions and controlare the responsibility of the driver. Further, these systems can beeither “active” or “passive” in how they are implemented. Active DASmeans that the vehicle itself controls various longitudinal and/orlateral aspects of the vehicle's driving behavior, or rather, veryspecific driving tasks, through its sensors, algorithms, processingsystems, and actuators. Passive DAS means that the vehicle will simplyassist the driver, through its sensors, algorithms, processing systems,and Human-Machine Interfaces (HMIs) with controlling variouslongitudinal and/or lateral aspects of vehicle control. For example, ina collision avoidance situation an active system would bring the vehicleto a stop or route the vehicle around the obstacle in the immediatepath. A passive system would provide some type of visual, auditory, andhaptic cues to the driver to stop or route the vehicle around theobstacle.

Thus, a DAS system helps the driver with many tasks ingrained into thedriving process and implemented specifically for the purpose to increasecar and road safety, as well as driver convenience. Such DAS systemsinclude, but are not limited to cruise control, Adaptive Cruise Control(ACC), active steering for lane keeping, lane change assistance, highwaymerge assistance, collision mitigation and avoidance systems, pedestrianprotection systems, automated and/or assistive parking, signrecognition, blind spot detection for collision mitigation, and stop andgo traffic assistance. Accordingly, the disclosed embodiments provideadditional and potentially more accurate data to such DAS systems toprovide this assistive functionality.

It should further be understood that disclosed embodiments utilizefunctionality from multiple different technological fields to provide anadditional mechanism and methodologies developing training data sets tofacilitate autonomous and/or assistive driving functionality bycombining analysis performed in computer vision and machine learning.

While the functionality of the disclosed embodiments and the systemcomponents used to provide that functionality have been discussed withreference to specific terminology that denotes the function to beprovided, it should be understand that, in implementation, the componentfunctionality may be provided, at least in part, components present andknown to be included in conventional transportation vehicles.

For example, as discussed above, disclosed embodiments use software forperforming functionality to enable measurement and analysis of data, atleast in part, using software code stored on one or more non-transitorycomputer readable mediums running on one or more processors in atransportation vehicle. Such software and processors may be combined toconstitute at least one controller coupled to other components of thetransportation vehicle to support and provide autonomous and/orassistive transportation vehicle functionality in conjunction withvehicle navigation systems, and multiple sensors. Such components may becoupled with the at least one controller for communication and controlvia a CANbus of the transportation vehicle. It should be understood thatsuch controllers may be configured to perform the functionalitydisclosed herein.

It should further be understood that the presently disclosed embodimentsmay be implemented using dedicated or shared hardware included in atransportation vehicle. Therefore, components of the module may be usedby other components of a transportation vehicle to provide vehiclefunctionality without departing from the scope of the invention.

Exemplary embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth, such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. In someillustrative embodiments, well-known processes, well-known devicestructures, and well-known technologies are not described in detail.

Terminology has been used herein for the purpose of describingparticular illustrative embodiments only and is not intended to belimiting. The singular form of elements referred to above may beintended to include the plural forms, unless the context indicatesotherwise. The method processes, and operations described herein are notto be construed as necessarily requiring their performance in theparticular order discussed or illustrated, unless specificallyidentified as an order of performance or a particular order isinherently necessary for embodiment to be operational. It is also to beunderstood that additional or alternative operations may be employed.

Disclosed embodiments include the methods described herein and theirequivalents, non-transitory computer readable media programmed to carryout the methods and a computer system configured to carry out themethods. Further, included is a vehicle comprising components thatinclude any of the methods, non-transitory computer readable mediaprogrammed to implement the instructions or carry out the methods, andsystems to carry out the methods. The computer system, and anysub-computer systems will typically include a machine readable storagemedium containing executable code; one or more processors; memorycoupled to the one or more processors; an input device, and an outputdevice connected to the one or more processors to execute the code. Amachine-readable medium may include any mechanism for storing ortransmitting information in a form readable by a machine, such as acomputer processor. The information may be stored, for example, involatile or non-volatile memory. Additionally, embodiment functionalitymay be implemented using embedded devices and online connection to cloudcomputing infrastructure available through radio connection (e.g.,wireless communication) with such infrastructure. The training datasets, image data, and/or transformed image data may be stored in one ormore memory module of the memory coupled to the one or more processors.

Although certain embodiments have been described and illustrated inexemplary forms with a certain degree of particularity, it is noted thatthe description and illustrations have been made by way of example only.Numerous changes in the details of construction, combination, andarrangement of parts and operations may be made. Accordingly, suchchanges are intended to be included within the scope of the disclosure,the protected scope of which is defined by the claims.

The embodiment(s) detailed hereinabove may be combined in full or inpart, with any alternative embodiment(s) described.

The invention claimed is:
 1. Transportation vehicle equipment fordeveloping transformed training data, the equipment comprising: at leastone processor; at least one sensor; at least one memory module, whereinthe transportation vehicle equipment gathers image data; wherein theimage data depicts a roadway and is stored on the at least one memorymodule; wherein the at least one processor analyzes the image data, todetect a horizon line and direction information for pixels representingthe roadway, and wherein the image data is generated using the at leastone sensor; and means for analyzing the image data that transforms theimage data depicting the roadway into transformed image data depictingan arcuate roadway, and means for developing a training data set fortraining a machine learning algorithm with the transformed image data.2. The transportation vehicle equipment of claim 1, wherein means fortraining trains at least a part of an autonomous and/or assistivedriving system for operating the transportation vehicle to travel on theroadway.
 3. The transportation vehicle equipment of claim 2, wherein themeans for developing the training data set for training the machinelearning algorithm replaces a portion of the image data with thetransformed image data.
 4. The transportation vehicle equipment of claim3, wherein the transformed image data represents synthetic curvedroadway segments.
 5. The transportation vehicle equipment of claim 1,wherein the at least one sensor is a camera disposed on thetransportation vehicle and communicatively coupled with the means foranalyzing the image data.
 6. The transportation vehicle equipment ofclaim 5, wherein the camera comprises predetermined extrinsicparameters.
 7. The transportation vehicle equipment of claim 5, whereinthe means for analyzing image data transforms the direction informationfor pixels representing the roadway from image coordinates to spatialcoordinates.
 8. The transportation vehicle equipment of claim 7, whereinthe means for analyzing image data determines a flat plane based on avertical distance of the camera from the roadway.
 9. The transportationvehicle equipment of claim 8, wherein the means for analyzing image datadetermines that the pixels disposed on the flat plane indicate a surfaceof the roadway.
 10. The transportation vehicle equipment of claim 9,wherein the means for analyzing image data determines distanceinformation for the pixels indicating the surface of the roadway. 11.The transportation vehicle equipment of claim 10, wherein the means foranalyzing image data applies a curvature to the pixels indicating thesurface of the roadway.
 12. A method of image transformation fortraining a machine learning algorithm, the method comprising: analyzing,by a processor, image data stored in a memory and depicting a roadwayacquired by one or more data gathering transportation vehicles to detecta horizon line and direction information for pixels representing theroadway, wherein the image data is generated using at least one sensormounted to the one or more data gathering transportation vehicles;transforming the image data depicting the roadway into transformed imagedata depicting an arcuate roadway; and developing a training data setfor training a machine learning algorithm with the transformed imagedata.
 13. The method of image transformation of claim 12, furthercomprising providing autonomous and/or assistive functionality foroperating a transportation vehicle to travel on the roadway, whichincludes transforming a vehicle path corresponding to the transformedimage data.
 14. The method of image transformation of claim 13, whereinthe operation of providing autonomous and/or assistive functionality foroperating the transportation vehicle to travel on the roadway furtherincludes replacing a portion of the image data with the transformedimage data.
 15. The method of image transformation of claim 14, whereinthe transformed image data represents synthetic curved roadway segments.16. The method of image transformation of claim 12, wherein the at leastone sensor is a camera disposed on the transportation vehicle andcommunicatively coupled with the processor and memory.
 17. The method ofimage transformation of claim 16, wherein the camera comprisespredetermined extrinsic parameters.
 18. The method of imagetransformation of claim 12, further comprising transforming thedirection information for pixels representing the roadway from imagecoordinates to spatial coordinates.
 19. The method of imagetransformation of claim 18, further comprising determining a flat planebased on a vertical distance of the camera from the roadway.
 20. Themethod of image transformation of claim 19, further comprisingdetermining that the pixels disposed on the flat plane indicate asurface of the roadway.
 21. The method of image transformation of claim20, further comprising determining distance information for the pixelsindicating the surface of the roadway.
 22. The method of imagetransformation of claim 21, further comprising applying a curvature tothe pixels indicating the surface of the roadway.
 23. The method ofimage transformation of claim 22, further comprising retaining originalcolor information in the image coordinated for pixels moved in thespatial coordinates during the transforming operation.
 24. Anon-transitory, machine readable medium including machine readablesoftware code, which, when executed on a processor, controls a method ofimage transformation for training a machine learning algorithm, themethod comprising: analyzing image data depicting a roadway gathered bya transportation vehicle to detect a horizon line and directioninformation for pixels representing the roadway, wherein the image datais generated using at least one sensor mounted to the transportationvehicle; transforming the image data depicting the roadway intotransformed image data depicting an arcuate roadway; and developing atraining data set for training a machine learning algorithm with thetransformed image data.
 25. The non-transitory, machine readable mediumincluding machine readable software code of claim 24, which, whenexecuted on a processor, controls a method of image transformation fortraining a machine learning algorithm, the method comprising: trainingthe machine learning algorithm with a portion of the image data and thetransformed image data.
 26. The non-transitory, machine readable mediumincluding machine readable software code of claim 24, which, whenexecuted on a processor, controls a method of image transformation fortraining a machine learning algorithm, the method comprising: executingthe machine learning algorithm to provide autonomous and/or assistivefunctionality for operating a transportation vehicle to travel on theroadway.